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ABSTRACT 



Contemporary educational research is marked by diversity, 
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the Methodologically Inclusive Research Synthesis (MIRS) , has been developed 
to offer a set of guidelines to facilitate informed choices at the critical 
decision points of any research synthesis process. The MIRS perspective has 
been developed on the premise/observation that methods of primary research 
and research synthesis hold a reflexive relationship. A dialectic tension 
exists between the methods of primary research and research synthesis where 
methodological developments at either level influence methodological 
developments at the other level . This tension is illustrated in the 
award-winning article "A Critical Analysis of the Research on Learning To 
Teach: Making the Case for an Ecological Perspective on Inquiry" (M. Wideen, 
J. Mayer-Smith, and B. Moon) which appeared in "Review of Educational 
Research" volume 68, number 2. This paper describes the essence of the MIRS 
framework by drawing illustrations from this article. (Contains 14 
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Essential Features of Methodologically Inclusive Research 
Syntheses 

Harsh Suri, The University of Melbourne 
Email: H.Suri@latrobe.edu.au 

(Paper presented as a part of the symposium, Primary Research, Secondary Research and 
Research Synthesis - Theory, Value and Linkage, at the Annual Meeting of the American 
Educational Research Association, New Orleans, April 1-5, 2002). 

Since the 1970s, several methods have been proposed to improve rigour in synthesising 
research across individual studies. Examples of some formally proposed methods of research 
synthesis include: “meta-analysis” (Glass, 1976), “best-evidence synthesis” (Slavin, 1986), 
“meta-ethnography” (Noblit & Hare, 1988), “exploratory case-study oriented review of 
multivocal literatures” (Ogawa & Malen, 1991), “cross-case analysis” (Miles & Huberman., 
1994), aggregated analysis (Eastabrooks, Field, & Morse, 1994), “qualitative meta-synthesis” 
(Sandelowski, Docherty, & Emden, 1997), “meta-analysis of qualitative research” (Jensen & 
Allen, 1994), “interpretivist-oriented reviews” (Eisenhart, 1998) or meta-study (Paterson, 
Thome, Canam, & Jillings, 2001). Each of these methods has its specific domain of 
applicability, strengths and weaknesses. 

I commenced my doctoral project with a quest for the most comprehensive method of research 
synthesis. To pursue this quest, I immersed myself into the literature on contemporary 
methods of primary research and research synthesis. As I delved deeper into the literature, I 
realised that no single method of research synthesis can be a best fit for all types of research 
synthesis to synthesise different types and combinations of primary research for a variety of 
purposes. After several modifications and refinements, the current quest of my doctoral 
project may be encapsulated in the following question. 

> Contemporary educational research is marked by diversity, complexity and richness 
of purposes, methods and perspectives. How can such variety and complexity be 
accommodated, encapsulated and celebrated at the level of synthesising educational 
research? 

My doctoral project proposes a set of guidelines to facilitate informed decision making at 
critical decision points of every trustworthy research synthesis process. I refer to these 
guidelines collectively as a “Methodologically Inclusive Research Synthesis ” (MIRS) 
framework or perspective. In this paper, I describe the essence of the MIRS framework. Also 
I demonstrate an enactment of these salient features by making references to an exemplary 
research synthesis. Since the 1978, the American Educational Research Association (AERA) 
has given the Review of Research Award in “recognition of an outstanding review of research 
article appearing in an AERA-sponsored publication” 

(http://www.aera.net/about/awards/rerpast.html) . In the year 2000, the award was given to 
Wideen, Mayer-Smith and Moon for their article in the Review of Educational Research 
(RER) titled A critical analysis of the research on learning to teach: Making the case for an 
ecological perspective on inquiry (Wideen, Mayer-Smith & Moon, 1998). I illustrate the 
essence of the MIRS framework by drawing illustrations from this award-winning review. 
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While developing the MIRS framework, I have been methodologically inclusive at four 
levels. First, I explored the possibilities of including a range of methodologically diverse 
primary research reports into research syntheses. Second, I explored the opportunities offered 
by adapting the techniques and perspectives from a variety of primary research methodologies 
to the process of a research synthesis. Third, I drew on ideas from several formally proposed 
methods of research synthesis as well as exemplary research syntheses. Fourth, I took into 
account varied interests of different stakeholders in educational research and practice. 

Problematising the Process of Research Synthesis 

Through the MIRS approach, I have attempted to problematise the entire process of a 
trustworthy research synthesis at a paradigmatic level. The common belief underlying 
different forms of formally proposed research syntheses is that evidence in educational 
research may be analysed and synthesised at three levels, each of which makes a worthy 
contribution to improve the practice of education. Within the MIRS framework, the 
methodological perspectives and techniques employed at each level of sense-making may be 
qualitative, quantitative or a combination of both. The form of research associated with each 
level is referred to as: primary research; secondary research; and research synthesis. My use 
of these terms are methodologically inclusive adaptations of Glass's (1976) definitions of the 
terms: primary analysis, secondary analysis and meta-analysis. I use the term meta-analysis as 
a broad umbrella term to refer to a variety of procedures associated with the entire process of 
statistical integration of primary research reports to make generalisable claims. I have 
refrained from re-defining the term “meta-analysis” in an inclusive sense as the term has been 
almost exclusively used in the past three decades to refer to “statistical analysis of the 
summary findings of many empirical studies” (Scope note for the term “meta-analysis” in 
ERIC Thesaurus). 

> Primary Analysis and Primary Research: Primary research involves going into the 
field or an experimental situation to collect raw evidence to pursue one’s own 
research question. The analysis or interpretation of this data is referred to as primary 
analysis and the individuals conducting such research are called primary researchers. 

> Secondary Analysis and Secondary Research: Secondary research involves re- 
analysing or re-interpreting raw evidence or data collected by other primary 
researchers for their own primary research. This re-analysis or re-interpretation could 
be pursued to address the same or a different question using different analytic tools or 
interpretive positions. 

> Research Synthesis: Research syntheses are different from primary research and 
secondary research in the sense that they analyse or interpret the analyses or 
interpretations reported by primary researchers rather than collecting, analysing or 
interpreting any raw data or evidence. 

These three levels of knowledge construction are complementary: neither adversarial nor 
hierarchical. Primary analyses and secondary analyses constitute the evidence for research 
syntheses. Research syntheses, in turn, can make explicit useful connections between 
individual primary or secondary reports to contribute towards a more comprehensive 
understanding of the field. Research syntheses can strengthen the relationship between 
educational research, policy and practice by disseminating succinct representations of primary 
and secondary research. Such complementarity between the purposes of primary research and 
research synthesis is also reflected in the goals of synthesis by Wideen, Mayer-Smith and 
Moon (1998). 
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First, we aim to establish what is currently known about how people learn to 
teach, and second, we will propose some direction for the focus and quality of 
future research and for the reporting of that research (p. 131). 

Nonetheless the “God’s eye view” that is sometimes attributed to research reviews should be 
contested with a call for greater transparency and accountability in the processes of research 
syntheses. A similar viewpoint is also expressed by Wideen, Mayer-Smith and Moon (1998). 

All researchers conduct research using a particular lens and that this focus has 
a bearing on what is seen, recognized as significant, and ultimately reported. 

This use applies to reviewers as well. Thus, we begin this review with a brief 
discussion of the perspectives we brought to the review and how they 
influenced the framing of the review and the review process (p. 131). 

Reflexivity Between the Methods of Primary Research and Research Synthesis 

The MIRS perspective has been developed on the premise/observation that methods of 
primary research and research synthesis hold a reflexive relationship. A dialectic tension 
exists between the methods of primary research and research synthesis where methodological 
developments at either level influence methodological developments at the other level. 

Methodological developments at the level of primary research shape the methods of research 
synthesis in two ways. First, primary research reports form the data of research syntheses. 
Changes in methods of primary research necessitate changes in methods of research synthesis 
Second, research synthesists often adapt appropriate techniques of primary research to 
different contexts of research synthesis. For instance, meta-ethnography was conceived as 
ethnographic research gained popularity in educational research. 

At the same time, rigour in methods of research synthesis also contributed to improving rigour 
in methods of primary research. For instance, meta-analysts have played an important role in 
raising awareness to improve rigour and reporting formats of statistical analyses in primary 
research reports. The MIRS approach is aimed at contributing to such collective efforts of 
promoting rigour by critical analyses of the current methodological trends in primary 
research. 

Celebration of such a reflexive relationship between the methods of primary research and 
research synthesis is also evident in different parts of the synthesis by Wideen, Mayer-Smith 
and Moon (1998). Throughout their synthesis, they substantiate their choices at the critical 
decisions points of their synthesis by adapting the techniques and perspectives that are well 
established in the arena of primary research methods. Also, they provide a critical and 
constructive commentary on the prevalent methods employed in the relevant primary research 
and propose suggestions on how to improve the rigour of future research studies in a way that 
will enhance their utility and trustworthiness. 



Methodological Inclusivity 

Making a case for the concept of a meta-analysis, Glass, McGaw & Smith (1981) observed: 
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Styles of research integration have been shaped by the size of the research 
literature. ... Integrating the research literature of the 1980s demands more 
sophisticated techniques of measurement and statistical analyses (p. 12). ... It 
is not uniformity in research reviewing and integrating that is desirable, rather 
it is clarity, explicitness, and openness - those properties that ... impart to 
inquiry its “objectivity” and trustworthiness (p. 20). 

Through the MIRS approach, I extend the concept of research synthesis beyond the 1980s. In 
my observation, styles of research synthesis have been shaped by the size and the styles of the 
relevant primary research literature. The Table below summarises the nature of typical 
primary research literature published on a given topic of research juxtaposed with the methods 
of research synthesis that were the norm in that period. In the 1940s and 1950s, primary 
research on a given topic of interest tended to be a relatively small number of primary 
research studies employing similar methods. These could be described aptly by adhoc 
narrative reviews that were a norm in education until the 1960s. In the 1960s and 1970s, 
primary research on many topics of interest tended to be a large number of studies that aimed 
to objectively report their findings on similar hypotheses. In the 1970s and 1980s, statistical 
methods of integrating findings across studies gained popularity in such domains of research. 
These methods could synthesise studies that addressed similar hypotheses with objectivity 
even if the individual findings varied in magnitudes and directions. By the 1980s and 1990s, 
primary research in education was distinctly marked by a diversity of purposes, perspectives 
and methods. The elusive nature of objectivity was increasingly being recognised/questioned 
along with a growing acceptance of qualitative research methods. A parallel drive for 
methodological inclusivity can also be observed in methods of research synthesis since the 
late 1980s. “Clarity, explicitness, and openness” are still upheld as canons of rigour. 
However, there is a growing recognition that these tenets may be pursued through a variety of 
quantitative and/or qualitative methods of sense-making. Such diversity of approaches to 
research synthesis is evident from the various formally proposed approaches as well as 
exemplar syntheses published during the last decade in the Review of Educational Research 
and Review of Research in Education. 



A Chronological Overview of Methodological Norms in Education 



Primary Research 


Research Synthesis 


1940s & 1950s: small number of 
employing similar methods 


Until 1960s: adhoc narrative reviews w< 
norm 


1960s & 1970s: large number of 
examining similar hypotheses 


1970s & 1980s: statistical methods of r< 
integration gained popularity 


1980s & 1990s: growing eclecticisn 

diversity of purpose, methods and perspect 


From late 1980s: growing eclecticist 
diversity of purpose, methods and perspect 



Contemporary primary research methods are marked by a growing acceptance of 
methodological diversity and eclecticism. Methodological inclusivity within the methods of 
research synthesis is essential to encapsulate such complexity of methods and findings typical 
of primary studies reported within most domains of educational research. Within the MIRS 
framework, methodological inclusivity is perceived as an important step to enhance the 
compatibility between the contemporary methods of primary research and research synthesis. 

A similar need for methodological inclusivity in educational research syntheses is also 
expressed by Wideen, Mayer-Smith and Moon (1998). 
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We used a broadly based and multimethod approach to prepare this review. In 
doing so, we followed the lead of Soltis (1984), who argued that the study of 
pedagogy requires the study of its empirical, interpretive, and normative 
dimensions. We, too, believe that any piece of relevant, quality research can and, 
furthermore, should contribute to our understanding of learning to teach. 



Purposefully Informed Selective Inclusivity 

Methodological inclusivity can be realised in numerous valid ways. Inclusion is always 
mirrored by exclusion. Thus, any form of inclusivity can only be partial inclusivity. Also, a 
rigorous research synthesis makes much more demands on time and resources when 
compared with adhoc reviews. The practical constraints of time, resources and access to 
information impact upon the methodological choices we make in a research synthesis just as 
they do in a primary research study. Through the MIRS approach, I hope to support 
purposefully informed selective inclusivity at every phase of research synthesis. 

A similar spirit of purposefully informed selective inclusivity is also illustrated in the review 
by Wideen, Mayer-Smith and Moon (1998). After describing the selection criteria and the 
search techniques employed their review along with the references to the previous reviews 
from which they have drawn heavily, they delimit the scope of their current review. 

Because a number of previously published reviews had dealt with the literature 
prior to 1990, we included, primarily though not exclusively, those studies 
published after 1990. Our selection criteria also excluded position papers and 
conceptual studies. We found that although there is a large body of literature on 
learning to teach, only a limited subset is based on primary data (p. 134). 

Through the MIRS framework, I present guidelines in the form of a set of critical decision 
points, various options at each decision point and the pros-and-cons of each option. These 
guidelines are aimed at helping synthesists in being wary of common sources of biases and 
errors. Also, they provide a framework for the readers of research syntheses to actively 
evaluate and adapt the information they read to their own context. Rather than prescribing 
rigid standards, I hope to foster critical awareness among varied stakeholders in producing 
and using research syntheses. 



Phases of a Research Synthesis 

Meta-analysts in education frequently endorse Cooper’s (1982; 1998) conceptualisation of an 
integrative research review as a scientific inquiry that involves five stages parallel to those of 
a primary research study. These stages of a meta-analysis are: problem formulation, data 
collection, data evaluation, analysis and interpretation, public presentation. The Handbook of 
Research Synthesis by the editors Harris Cooper and Larry V. Hedges (1994) draws from the 
experience of 43 major proponents of meta-analysis to discuss in detail the issues and 
methodological choices, associated with each of these stages, in the context of a meta-analytic 
integrative review. Through the MIRS approach, I have attempted to extend this discussion by 
identifying the phases in the context of a research synthesis framework that is more inclusive 
of diverse methodologies. 

I began by unpacking each stage of a meta-analysis in the light of the insights gained through 
my interactions with colleagues and published literature on a wide range of methods of 
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primary research and research synthesis. This involved unravelling the associated 
euphemisms and problematising embedded assumptions with the purpose of making it more 
transparent and methodologically inclusive. As meta-analytic stages were conceptualised in 
the context of quantitative research, I drew heavily from qualitative and mixed primary 
research methods texts to adapt Cooper’s stages of research synthesis for a methodologically 
inclusive context. After brainstorming a wide range of questions relevant to the processes of a 
wide range of research syntheses, I grouped these questions into six broad clusters of tasks. 
These clusters were refined further into the following six phases of a research synthesis: 
framing the purpose and orientation; selecting the evidence; distilling the information; 
constructing connected understanding(s); sharing with an audience; and evaluating a research 
synthesis. 

> Framing the purpose and orientation. A research synthesis may serve one or more 
diverse purpose to voice and/or challenge the concerns of different stakeholder in the 
production and use of educational research. These purposes may be pursued from a 
variety of theoretical and methodological orientations. 

> Selecting the Evidence. Primary research reports form the evidence of a research 
synthesis. The selection criteria and the search strategies employed in the synthesis 
may be chosen in alignment with its purpose and orientation. 

> Distilling the Information. A variety of techniques and criteria may be employed to 
extract, interpret and evaluate the relevant information from individual studies. 

> Constructing Connected Understandings. The relevant information extracted from 
individual studies may be connected through various lenses and strategies to construct 
understandings of different aspects of the phenomenon. 

> Sharing with an audience. The overall structure, tools and techniques of the synthesis 
report may be selected to match the intended audience and the impact that the 
synthesist wishes to make through the synthesis. 

> Evaluating a research synthesis. The general criteria that may be employed to 
evaluate every research synthesis include: coherence between the purpose and the 
process; reflexivity between the process and the product; and transparency at every 
critical decision point of the process. The specific techniques for achieving these 
general criteria may be chosen in the light of the different aspects of the synthesis 
design. 

In practice, these phases are likely to overlap with tasks involved in more than one phase 
being carried out simultaneously. Considerations within each phase may inform and refine the 
process of other phases. The sequential nature of these phases is preferably conceptualised as 
a spiral, rather than a linear, process where each phase tends to be revisited and refined 
several times. The verbs associated with each phase are presented in present continuous tense 
to emphasise the ongoing nature of each phase and refrain from implying a sense of finality. 
Within each phase, a sense of objective distancing is avoided by emphasising the role of the 
synthesist as a sensitive human instrument. 

Explicit Reflexivity Between the Process and the Product 

The product of any research synthesis is influenced by the critical decisions made by the 
synthesist in the process of the synthesis. In practice, the choices made during the process of a 
research synthesis are guided by the nature of the intended purpose of the synthesis product. 
In a rigorous synthesis, the synthesist iteratively reflects upon the intended purpose of the 
product and maps the process accordingly. Also, the synthesist reflects upon how the critical 
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decisions made in the process may shape the final product of the research synthesis. 
Synthesists may deliberately make certain methodological choices to pursue one purpose that 
would be made differently for another purpose. Likewise, certain methodological decisions 
will provide particular insights into a phenomenon that may be different from those found 
through alternative methodological decisions. Through the MIRS approach, I have attempted 
to support such reflexivity between the process and the product of a research synthesis. Such 
methodological awareness promotes critical thinking in the production and use of research 
syntheses. 

Such an explicit reflexivity between the synthesis process and the synthesis product is evident 
throughout the synthesis by Wideen, Mayer-Smith and Moon (1998) as illustrated through the 
following excerpt. 

We begin this review with a brief discussion of the perspectives we brought to 
the review and how they influenced the framing of the review and the review 
process (p. 131). 



Transparency to Improve Transferability 

Through the MIRS approach, I have emphasised the need for a transparency of the process 
within research synthesis reports to enhance the transferability of the product. Increased 
transparency of the process would require the research synthesist to delineate and substantiate 
the critical choices made while conducting a synthesis. Also, research synthesists would share 
their reflections on the possible consequences that their decisions may have had in shaping the 
synthesis products. Transparency of process would allow readers to evaluate critically the 
level(s) of (dis)similarities of the synthesis context or standpoint with their own contexts or 
viewpoints. This, in turn, can improve informed transferability of the product: the readers can 
adapt the synthesis product in varying degrees to their own contexts. 

Wideen, Mayer-Smith and Moon (1998) also recognise the need for a transparency in the 
process of a synthesis. 

We recognize that, as a research team, we bring our own biases to this review, 
which may surface in the pages that follow. We can claim only that we have 
attempted to reduce the influence of our biases by making our perspectives 
known to each other and to the readers of this review, where possible (p. 132). 



Conclusion 

Contemporary educational research is marked by diversity, complexity and richness of 
purposes, methods and perspectives. The MIRS framework is conceptualised on the premise 
that such variety and complexity should also be celebrated at the level of synthesising 
research. The MIRS framework offers a set of guidelines to facilitate informed choices at the 
critical decision points of any research synthesis process. This paper describes the essential 
features of the MIRS framework by drawing on illustrations from an exemplary synthesis. 
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